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Project  Summary 


•  Objectives 

-  Synthesize  eognitive  theory  of  eollaboration 

-  Artieulate  eritieal  knowledge  for  sueeessful  eollaboration 

-  Develop  eognitive-based  eollaboration  metries 

-  Apply  theory  to  help  teams  eollaborate  more  effeetively  (foeus  of  this  presentation) 

•  Research  Questions 

-  Critical  knowledge  for  effective  collaboration 

-  Means  to  package  awareness  of  critical  knowledge  to  improve  collaboration 

-  Means  to  leverage  cognitive  framework  to  identify  collaboration  metrics 

•  Project  Status 

-  Third  and  final  year  of  project 

-  Metrics  summarized  in  handbook  “Command  Center  Performance  Assessment 
System” 

-  Five  teams  have  tried  out  the  Collaboration  Advizor  tool.  All  agreed  it  provided 
appropriate  diagnosis,  that  its  recommendations  were  useful,  and  that  it  helped  teams 
surface  and  discuss  critical  issues 

-  Formal  evaluation  planned  at  Navair  in  February 

-  Commercial  version  of  tool  under  development 

-  Metrics  and  tool  transitions  on  track 
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Topics 


•  Why  teams  fail 

•  Project  goals 

•  Theory 

•  Diagnosis  and 
recommendation  tool 

•  Chronology,  validation, 
publications,  and 
transitions 

•  Summary 


Why  T earns  F ail 


•  Physical  causes 

-  The  goal  is  too  hard  for  any  team  to  achieve,  given  the 
available  resources 

•  Cognitive  causes 

-  The  goal  is  not  too  hard  to  achieve,  but  the  team 
doesn’t  know  how  to  achieve  it 

•  Social  causes 

-  The  goal  is  not  too  hard  to  achieve  and  the  team  knows 
how  to  achieve  it,  but  team  members  are  not  willing  to 
work  hard  enough 
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Examples: 

Team  Failures  for  Cognitive  Reasons 


The  Bay  of  Pigs,  1962 

A  talented  and  intelligent 
policy  team,  but  Groupthink 
doomed  the  team  to  an 
unworkable  plan  with 
disastrous  results 


The  Iranian  airline  shootdown, 
July  3,  1988 

Well  trained  team  on  Vincennes,  but 
misunderstandings  of  each  others 
information  and  perspectives  led  to  a 
tragic  mistake 


Goals 

Help  Teams  Do  Well  Cognitively 


•  Problems 

-  There’s  a  morass  of  cognitive  factors 

-  It’s  hard  to  understand  cognitive  factors 

-  It’s  hard  to  diagnose  specific  problems 

-  It’s  hard  to  determine  effective  means  to  address  the  problems 

•  Approach 

-  Synthesize  cognitive  theory  of  collaboration 

-  Organize  cognitive  issues  to  help  teams  understand  them  (e.g.,  create 
periodic  table  of  cognitive  factors) 

-  Create  a  tool  to  help  team  members  diagnose  problems  and  identify 
remedies 

-  Validate  tool  effectiveness 

-  Explain  and  publicize  cognitive  team  issues 


Theory 


•  Basic  premises 

•  Collaboration  building 
blocks 

•  The  twelve  enablers 
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Premises 

Knowledge  Basis  for  Collaboration 


•  Knowledge  is  central  to  collaboration  and  teamwork 

-  Teams  whose  members  know  what  they  need  to  know  ean  work  together  effeetively. 

Those  that  do  not  are  prone  to  various  kinds  of  predietable  errors,  with  the  type  of  error 
dependent  on  the  type  of  knowledge  defieieney 

•  Knowledge  must  be  distributed  among  members  of  a  team 

-  Everybody  does  not  need  to  know  everything  for  a  team  to  be  effeetive.  But  every  team 
member  does  need  to  know  how  to  get  the  knowledge  he  or  she  needs. 

•  Individuals  need  to  know  about  both  “taskwork”  and  teamwork 

-  Taskwork  knowledge  is  team  members  need  to  earry  out  their  tasks  were  they  aeting  alone 

-  Teamwork  knowledge  is  what  team  members  need  to  know  to  work  together  effeetively 

•  The  collaborative  dialog  helps  generate  the  needed  teamwork  and  taskwork 
knowledge 

-  Team  members  exehange  ideas  to  elarify  issues  and  reaeh  eonsensus  to  put  in  plaee  the 
knowledge  and  understandings  that  team  members  must  have  to  aehieve  the  team’s 

mission. 
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Theory:  How  Teams  Work 

uilding  Blocks  of  Collaboration  and  Teamwork 


Team  Set  Up  and 
Adjustment 

•  Form  team 

•  Review  goals 

•  Identify  tasks 

•  Determine  roles 


Need  for  Team 

ehanges  set  up 


Group  Problem 

Synchronize 

Solving 

and  Act 

•  Brainstorm 

•  Mass  effects 

•  Prioritize 

•  Lay  groundwork 

•  Diseover  differenees 

•  Hand  off  tasks 

•  Enrich 

•  Backup 

•  Reach  consensus 

•  Cue  to  situation 

A 

i 

Issues  to 
work  on 

Diseussion 

results 

Performance 

feedback 

What  to 
do  next 

Individual  and  Shared  - 

Understandings 

•  About  plan,  goals,  tasks,  and  situation 

•  About  team  members  baekgrounds, 
aetivities,  and  status 

•  About  task  and  team  status  and  prospeets 


Group  Problem  Solving 

Sharing  of  Perspectives 


•  Collaboration  enables  teams  to  “make  better  lists” 

-  Better  views  on  what  is  happening,  the  reasons  for 
these  occurrences,  and  their  impacts  on  the  team 
mission 

-  Better  set  of  candidate  actions  to  take  in  response  to 
these  impacts 

-  Better  set  of  criteria  to  consider  when  evaluating  the 
desirability  of  these  actions 

-  Better  estimates  of  possible  consequences  of  the 
alternatives  being  considered 
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Information  to  Product 


Information 


Knowledge 

Risks: 


Needed  Team 

Effective 

Member 

— ► 

Team 

Knowledge 

◄ . 

Behaviors 

Importance 

Multipliers 


Behavioral 

symptoms 


Product  Quality 
or  Action 
Effectiveness 


Factors*  that 
increase 
difficulty  of 
obtaining 
needed 
knowledge 


Factors**  that 
increase 
importance  of 
obtaining 
needed 
knowledge 


Reflections  of 
knowledge 
inadequacies*** 


*  asynchronous,  geographically  distributed, 
cultural  diversity,  heterogeneous 
knowledge,  unique  roles,  task 
dependencies,  rotating  team  members 


**highly  interdependent  tasks, 
poor  management  support, 
intelligent  adversary,  dynamic 
environment 


***missed  deadlines,  failure  to  offer 
help,  surprise  at  others  behaviors, 
failure  to  share  information, 
redundant  or  missed  tasks 
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The  Twelve  Enablers 


•  Represents  basie  cognitive  foundations  for 
effective  collaboration 

•  At  a  level  useful  for  diagnosis  and 
recommendations 

-  Deficiencies  in  enablers  are  the  underlying  causes  of 
teamwork  problems 

-  Risks  and  symptoms  map  easily  to  enablers 

-  Recommendations  follow  directly  from  them 

•  Generalizes  well  known  critical  C2  and  decision 
functions  for  teams 
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Knowledge  Enablers 

Foundational  Knowledge 


©  Goals 


Others 


Plans 


Business  rules 


Dependencies 


Task  skills 
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Knowledge  Enablers 

Real  Time  Understanding  and  Assessments 


Activity 

Awareness 


Mutual 

Understanding 


External 

Situation 


Plan 

Prospects 


Task 

Progress 


Decision 

Factors 
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Principal  Enabler  Dependencies 
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Diagnosis  and  Recommendation  Tool 


•  Expert  system 
software 

-  Educates 

-  Alerts  to  possible 
knowledge  problems 

-  Warns  of  consequences 

-  Shows  areas  of 
agreement/ 
disagreement 

-  Suggests  ways  to 
improve 
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Tool  Modes 


•  Individual  mode 

-  Asks  questions  about  the  team,  task,  and  environment 

-  Collects  answers 

-  Evaluates  enablers  that  matter  and  are  at  risk  or  deficient 

-  Helps  people  explore  issues  individually 

•  Team  view 

-  Assembles  answers 

-  Shows  areas  of  agreement  and  disagreement 

-  Provides  safe  discussion  forum 

•  Trends  view 

-  Changes  in  team  knowledge  enablers  over  time,  with  critical 
events  superimposed 

-  Changes  to  risks  and  symptoms  responsible  for  enabler  changes 
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Storyboard  for  Commercial 
Version  to  Tool 


upon  completing  use  of  the  tool,  the  user  will  see  a  screen  presenting  several 
options: 


September  2003 
October  2003 
November  2003 
December  2003 
January  2004 
February  2004 


TEAM  RESULTS 


MOST  RECENT  TESTING  -  NOV.  28.  2003 
NUMBEROFTEAM  MEMBERS-  15 


Enablers 


Questions 


Goals 


Recommend 


Help 


Return  to  Menu 


Enabler  Areas  of  Concern 

Knowledge  Strength - 


£ 

Planning 

9 

Dependencies 

Each  Other 

9 

Business  Rules 

@ 

Task  Skills 

ActlvHy  Awareness 

External  Situation 

Task  Progress 

0 

Mutual  Understanding 

O 

Plan  Assessment 

® 

Decision  Factors 

weak 


strong 


Description: 


Description  of  Planning 


It  is  important  to  understand  the  team’s 
iina  tfisks,  schedules,  resources,  and 
success.  I  more  l 


This  is  the  overall  understanding  of  the  plan,  which 
should  define  tasks,  assign  those  tasks  to  team 
members,  and  specify  a  work  schedule. 

Plans  should  also  outline  leadership  roles,  identify 
individuals  as  subject  matter  experts,  and  clarify 
who  is  responsible  for  a  task  and  who  will  support 
that  person’s  efforts. 


Click  to  close 


I  is  are  weak  in  Planning,  team 
nay  work  on  others’  tasks  or 
h  iir  own  tasks.[  more  ] 


e  ement  over  responsibilities, 
t  ;d  tasks. 

c  ant  efforts.  ( more  ] 


omplex  project  plan, 
ed  roles  and  responsibilities. 
■I  ly  team  members.  [  more  ] 


}  ions: 


sks  and  deadlines. 


2.  Write  out  the  plan. 

3.  Hold  regular  meetings.  [  more  ] 


•  requires  immediate  attention  -  requires  monitoring  Q  -  acceptable 


TEAM  RESULTS 


MOST  RECENT  TESTING  -  NOV  28.  2003 
NUMBEROFTEAM  MEMBERS'  15 


Enablers 


Questions 


Recommend 


Help 


Return  to  Menu 


Questions  for:  Enabler  -  Planning 


H  High  Agreement:  There  is  a  probiem. 

Poor  Agreement:  There  may  be  a  problem. 
High  Agreement:  There  is  not  a  problem. 


Questions 


I  The  team 
roles  sch 
resource: 
goals. 


All 


Discussion 
Education 
Symptoms 
Risks 

Recommendatioi 
Agreement 

^The  team 

make  sure  that  all  team  memi 
understand  and  support  the 

3.  The  team  never  clearly  define 
member  roles  and  task 
responsibilities. 

4.  The  team  never  clearly  specif 
documented  the  schedule,  in 
task  deadlines  and  intermedi< 
milestones. 


Agree 

A  T 


Disagree 

A  ▼ 


Enablers 


Planning 


Recommendation 


All 


Recommendation 

1 .  Articulate  the  team’s  goals,  desired  products, 
and  how  to  achieve  both. 

2.  Diagram  the  tasks  that  support  common 
goals  and  that  occur  together  or  sequentially. 

3.  Create  a  timeline  of  task  milestones. 

4.  Create  a  workflow  chart  that  describes  tasks, 
dependencies,  resource  needs,  and 
individual  assignments. 

Click  to  close 


T. 


|Team  members  do  not  know  what 
resources  have  been  assigned  to 
tasks. 


4 

1 

Planning 


ti|on 

anding 
^nt 
s 


ie  team’s  goals  and 
ducts.  [  more  j 


iditions  for  success. 


requirements.  [  more  ] 


ustomers.  [  more  ] 


Describe  necessary  tasks.  [  more  ] 


L. 


More  Results  Print  Screen 


TRENDS  OVERTIME 


TESTING  DATES  -  SEPT  03  THRU  |AN  '04 
NUMBEROFTEAM  MEMBERS'  15 


Enablers 


Questions 


Help 


Return  to  Menu 


Trends  of  Knowledge  Strength  Over  Time 


cs 

1  Goals 

.  1 

Planning 

0 

1  Dependencies 

£ 

6 

1  Each  Other 

1  Business  Rules 

1  Task  Skills 

Activity  Awareness 

ii 

1  External  Situation 

Task  Progress 

d 

1  Mutual  Understanding 

Q 

1  Plan  Assessment 

S 

1  Decision  Factors 

■ 

view  All  Enablers 

view  the  Most  Improved 
- - 

View  the  Least  Improved 


strong 


Knowledge 

Strength 


This  is  a  composite  view  of  the  team's  progress  in  terms  of  the  Planning 
Enabler.  This  composite  integrates  all  of  the  data  from  the  team's  input  on 
questions  related  to  Planning. 

_ J 


I  Go  to  Details 


I  Print  Screen 
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Development  Chronology 


•  2000  (Phase  1):  Metrics,  theories,  taxonomies  (with  Klein) 

•  2001  (Phase  2  begins):  Case  studies,  initial  enabler  set,  paper-based 
guidelines 

•  Fall  2002:  Version  1.  Medical  diagnosis  model,  expert  system 
framework,  individual  mode  only 

•  Winter  2003:  Version  2.  War  Room  and  CCRP  team  use,  add  team 
view,  demo  at  Navy  Opportunity  Forum 

•  Summer  2003:  Version  3.  Improved  question  selection,  with 
emphasis  on  exploring  issues.  NPGS  team  tryout.  Define  trend  view 

•  Fall  2003:  Version  4.  Improved  knowledge  base  increasing  focus  on 
education.  UK  team  evaluation.  Planned  Navair  evaluation.  Tool 
patent  submission 

•  Spring  2004.  Commercial  version 
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V  alidation 


•  From  existing  literature 

-  Consistency  of  theory,  risks,  and  symptoms  with  academic  and 
business  management  findings 

•  ‘Taee  value”  validation  of  the  tool 

-  Five  teams  tried  it  out:  EBR  war  room  (six  times)  and  CCRP 
teams,  NPGS  teams  (twice),  and  two  UK  teams 

-  Team  feedback:  diagnoses  appropriate  and  helped  identify  real 
issues,  useful  recommendations 

•  Formal  experiments 

-  Piggy  backing  on  Navair  in  February 

-  Will  test  statistically  if  teams  with  tool  do  better  than  teams 
without  the  tool 

-  Streamlined  tool  to  focus  on  educational  utility 
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Lessons  Learned 


•  Cognitive  framework  of  twelve  enablers  makes  sense  to  users 

•  Paper-based  guidebook  for  diagnosis,  discussion,  and 
recommendations  too  hard  to  use. 

•  Expert  system  version  easy  to  use.  Users  perceived  diagnosis  to  be 
accurate  and  helpful 

•  Tool  perceived  to  contribute  in  three  ways 

-  Educational:  Informs  users  of  critical  knowledge  and  processes 

-  Discussion:  Provides  a  socially  safe  forum  for  raising  issues 

-  Recommendations:  Provides  good  checklist 

•  Leaders  may  feel  threatened  by  Advizor  tool 

•  Tool  cannot  ignore  social  issues  entirely,  but  it’s  risky  to  address 
too  obviously  and  directly 

•  May  need  to  tailor  language  and  emphasis  of  tool  for  different  types 
of  teams 
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Some  Publications 


•  Conference  Proceedings 

-  “Cognitive-Based  Metries  to  Evaluate  Collaboration  Effeetiveness.”  SAS  039 
Symposium:  Analysis  of  Military  Effeetiveness  of  Future  C2  Coneepts  and 
Systems,  April  2002 

-  Collaboration  Advizor™  Diagnosing  and  Fixing  Collaboration  Problems  Navy 
Opportunity  Forum.  May  2003 

-  “Understanding  and  Applying  the  Cognitive  Foundation  of  Effeetive 
Collaboration”  1 5^^  International  Conferenee  on  Systems  Researeh,  Informative, 
and  Cyberneties 

-  “Objeetive  Metries  For  Evaluation  of  Collaborating  Teams”  2003  Command  and 
Control  Researeh  and  Teehnology  Symposium,  Info  (June  2003) 

-  “Understanding  and  Applying  the  Cognitive  Foundation  of  Effeetive 
Collaboration”  5*  Annual  ONR  Conferenee  on  Collaborative  Deeision-Support 
Systems,  Quantieo,  VA,  September  1 1,  2003 

•  Other 

-  Command  Center  Performanee  Assessment  System,  Evidenee  Based  Researeh, 
April  2003 

-  Government  Exeeutive  “The  Cognitive  Path  to  Team  Sueeess.”  (submitted) 

-  Navy  Institute  Proeeedings  “Network  Centrie  Teamwork”  (in  draft) 

-  DTIC  “Cognitive  Basis  of  Collaboration”  (in  draft) 
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Transitions 


•  Collaboration  Diagnosis  and  Recommendation  Tool 

-  Currently  used  in  EBR  war  room 

-  Being  evaluated  by  war  room  clients  for  internal  use  and  by 
Collaborative  Information  Environment,  JFCOM 

-  To  be  demonstrated  to  Center  for  Naval  Leadership,  Little  Creek 

•  Collaboration  evaluation  and  evaluation  metries 

-  Support  to  JFCOM  experimentation 

•  Commereial  book  on  eognitive  basis  of  effeetive 
eollaboration 

-  Will  follow  publication  path  successfully  employed  by  War  Room 
book 
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Summary 


•  A  knowledge-centered  theory  can  explain  how  collaboration 
works 

•  This  theory  informs  collaboration  performance  metrics, 
guidelines  for  diagnosing  collaboration  problems,  and  methods 
for  correcting  collaboration  problems 

•  Metrics  documented  and  considered  for  use  at  JFCOM 

•  Five  teams  have  tried  out  and  used  the  Collaboration  Advizor 
tool 

-  Felt  set  of  twelve  cognitive  enablers  made  sense 

-  Said  tool’s  diagnoses  were  accurate  and  helpful 

-  Agreed  tool  helps  by  educating  team  members  on  important 
cognitive  factors,  by  surfacing  areas  of  agreement  and 
disagreement,  and  by  suggesting  remedies 

•  Commercial  version  of  tool  to  be  completed  this  spring 

•  Transitions  and  publications  on  track 


